feat(indexer): add Prometheus /metrics endpoint with indexer and transfer counters - #175
Conversation
…sfer counters
Wraith runs as a persistent background service with no operational metrics: a
stalled indexer or a degrading RPC endpoint was visible only in the logs.
New src/metrics.ts registers five custom metrics into a module-local registry
(not prom-client's process-global default, which is shared state no test can
clear safely) alongside the standard process/Node collectors:
ledgers_indexed_total counter {network}
transfers_stored_total counter {network, type}
rpc_errors_total counter {outcome}
last_indexed_ledger gauge {network}
db_query_duration_seconds histogram {operation}
The ledger counter takes the per-poll delta rather than the absolute sequence,
so a process resuming from a DB cursor does not report millions of ledgers
indexed in one second on every restart. rpc_errors_total counts attempts, not
calls: withRetry hides transient failures by design, so per-call counting would
read zero right up until the indexer falls over.
GET /metrics reads in-process counters only — no DB, no RPC — so it keeps
answering while the subsystems it reports on are down, and it is exempt from
both the rate limiter and the stale-read RPC probe for the same reason.
/status gains last_indexed_ledger as a snake_case alias of lastIndexedLedger,
matching the gauge name; the existing field is untouched.
|
@royalTreasure Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits. You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀 |
Miracle656
left a comment
Miracle656
left a comment
There was a problem hiding this comment.
Approved and merging. This is the best-instrumented PR in the wave — it reads like it was written by someone who has been paged before.
Three decisions I want to call out, because each one is the non-obvious choice and each is right:
The ledger counter takes the delta, not the sequence. A resumed process starts from wherever the DB cursor left it. Feeding the absolute sequence into a counter would report several million ledgers indexed in one second on every restart, which poisons rate() and makes the one alert you actually want — a flat rate meaning the loop has stalled — unusable. Guarding on advanced > 0 also means a backwards cursor contributes nothing rather than a negative.
rpc_errors_total counts attempts, not calls. This is the subtle one. withRetry exists to hide transient failures from callers, so a per-call counter reads a clean zero right up until the indexer falls over — the metric would be silent for exactly the window you needed it. Counting attempts, split retry / exhausted, surfaces a degrading endpoint while it is still succeeding. That distinction is the whole value of the metric.
/metrics is exempt from both the rate limiter and the stale-read RPC probe, and touches neither DB nor RPC. A scrape endpoint that 429s under load goes blind exactly when the graphs matter, and one that makes a network round-trip per scrape reports on an outage by participating in it. The test does not depend on the database or RPC being up pins this by failing both mocks and asserting 200 — that is the right test to have written.
Also correct: a module-local Registry instead of prom-client's process-global default, and timing failed queries in observeDbQuery's finally — the eight-second query that then throws is the one worth seeing, and dropping it would leave a histogram that only describes the healthy path.
Verified locally on a merge with main: tsc --noEmit clean, full suite 299/299.
One follow-up, not a blocker: npm flags prom-client as deprecated in favour of @prometheus-io/client. 15.1.3 is stable and universally used, so this is fine to ship — I'll file a note to track the rename.
Thanks — this raised the bar.
Both conflicts were additive unions: - src/api.ts: Miracle656#175's metrics import beside Miracle656#163's network middleware import. - README.md: Miracle656#175's /metrics section beside Miracle656#163's /readyz section, with both /status notes kept under the /status example. openapi.json regenerates identically from src/openapi/build.ts after the merge, so the committed document is not stale.
Derived per-token balances for an address, summing what it received and subtracting what it sent across the indexed history. Rebased onto main, which has moved a long way since this branch: the token cache landed via Miracle656#46 and the metrics module via Miracle656#175, so those parts of this PR are dropped as duplicates and what remains is the balance endpoint itself. Three fixes to the query on the way in: - The table reference was unqualified, `FROM "TokenTransfer"`. Every other raw query in db.ts uses `"wraith"."TokenTransfer"`, because the models declare @@Schema("wraith") — unqualified it resolves only if search_path happens to include the schema, so it would work locally and fail on a deployment that sets search_path differently. - No network predicate. Summing both chains' transfers for one address gives a number that corresponds to no balance anywhere. Now takes the network and filters on it, with the route reading it from the selector so an unknown network 400s instead of silently answering for the default. - The metrics timer was started and stopped around the query but not in a finally, so a throw leaked it. Uses observeDbQuery, which times failures too — a query that takes eight seconds and then fails is the one worth seeing. Mounted on the existing accounts router rather than a second one, so it sits beside /summary and /transfers and inherits the network middleware. The response keeps this PR's honesty about what the number is — a sum over the indexed window, not an on-chain read — and returns both the raw stroop amount and the display string, so a consumer doing arithmetic does not have to parse the decimal back and guess the scale. tsc clean; full suite 402 passed.
* feat: implement tiered token metadata caching with Prisma persistence and RPC fallback * feat: implement Prometheus metrics collection with registry and endpoint testing * feat: implement accounts balance route with ledger-derived token balances * chore: add vitest as devDependency for test:integration * fix: exclude broken upstream tests from jest (opa, integration) * Add GET /accounts/:address/balance, network-scoped and schema-qualified Derived per-token balances for an address, summing what it received and subtracting what it sent across the indexed history. Rebased onto main, which has moved a long way since this branch: the token cache landed via #46 and the metrics module via #175, so those parts of this PR are dropped as duplicates and what remains is the balance endpoint itself. Three fixes to the query on the way in: - The table reference was unqualified, `FROM "TokenTransfer"`. Every other raw query in db.ts uses `"wraith"."TokenTransfer"`, because the models declare @@Schema("wraith") — unqualified it resolves only if search_path happens to include the schema, so it would work locally and fail on a deployment that sets search_path differently. - No network predicate. Summing both chains' transfers for one address gives a number that corresponds to no balance anywhere. Now takes the network and filters on it, with the route reading it from the selector so an unknown network 400s instead of silently answering for the default. - The metrics timer was started and stopped around the query but not in a finally, so a throw leaked it. Uses observeDbQuery, which times failures too — a query that takes eight seconds and then fails is the one worth seeing. Mounted on the existing accounts router rather than a second one, so it sits beside /summary and /transfers and inherits the network middleware. The response keeps this PR's honesty about what the number is — a sum over the indexed window, not an on-chain read — and returns both the raw stroop amount and the display string, so a consumer doing arithmetic does not have to parse the decimal back and guess the scale. tsc clean; full suite 402 passed. --------- Co-authored-by: Miracle656 <iupacnumen2020@gmail.com>
2 participants
Closes #39
Summary
Wraith runs as a persistent background service with no operational metrics — a stalled indexer or a degrading RPC endpoint was visible only in the logs. This adds a Prometheus scrape endpoint and instruments the indexer, the RPC retry path, and the hot database operations.
Metrics
New
src/metrics.ts, usingprom-client:ledgers_indexed_totalnetworktransfers_stored_totalnetwork,type(fungible/nft)rpc_errors_totaloutcome(retry/exhausted)last_indexed_ledgernetworkdb_query_duration_secondsoperationStandard
process_*/nodejs_*collectors are registered alongside them.Two details worth calling out:
rpc_errors_totalcounts attempts, not calls.withRetryhides transient failures from its callers by design, so counting only calls that exhausted their retries would read zero right up until the indexer falls over.outcomekeeps the two readings separable.Everything registers into a module-local
Registryrather than prom-client's global default — the global is process-wide state shared with any dependency that also uses prom-client, and it cannot be cleared between tests without clobbering theirs.Endpoint
GET /metricsreturns the exposition format with prom-client's content type. It reads in-process counters only — no DB, no RPC — so it keeps answering while the subsystems it reports on are down, which is the point of having it. For the same reason it is exempt from:Registered in the OpenAPI document (
src/openapi/build.ts,openapi.jsonregenerated vianpm run docs:openapi).Instrumentation
src/indexer.ts— ledger progress and stored-row counts on both the single-poll and the parallel (INGEST_WORKERS > 1) paths, including the empty-batch case where the cursor still advances.src/rpc.ts—withRetryrecords each failed attempt.src/db.ts—observeDbQuerywrapsupsertTransfers,upsertNftTransfers,getLastIndexedLedger,setLastIndexedLedger, andqueryTransfers. Failures are timed too: a query that runs for eight seconds and then throws is exactly the one worth seeing on a latency graph./statusNow reports
last_indexed_ledgeralongside the existinglastIndexedLedger, matching the gauge's name. It is an additive alias — the camelCase field is untouched and both always carry the same value.Tests
src/__tests__/metrics.test.ts(8 tests): valid exposition format and content type, all five custom metrics present with the correct# TYPE, recorded samples rendered with their labels, the endpoint still serving 200 while DB and RPC both reject,observeDbQuerytiming both the success and the throwing path (and rethrowing), and/statuscarrying both ledger field names.Verification
npm test— 298 passed across 27 suites (was 290/26); coverage thresholds still met.npx tsc --noEmit— clean.npm run docs:openapi— regenerated, diff is the/metricspath only.